[Draft][MoE][Offload] Row-level expert weight cache provider (moe_expert_cache_provider=row) by 01554 · Pull Request #47 · 01554/vllm

01554 · 2026-09-09T02:08:25Z

Upstream extraction, unit 3 (notes/upstream-expert-provider-contract-draft-2026-09-09.md). Draft; two reviews required before ready (astra_mid: review 1, astra: review 2 incl. GPU).

Base: lab/upstream-37190-base d38903c = upstream/main a97dacb + PR vllm-project#37190 c66868c (notes/upstream-37190-base-merge-2026-09-09.md). Head: 60ba5b9 (last GPU-verified production state: b5ac54e; 60ba5b9 adds docstrings and moves one log line). Dependencies: vllm-project#37190 (provider surface). The native consumer path (not in this PR) additionally depends on #44.

Commits on this head

Knob: OffloadConfig.moe_expert_cache_provider (cached default, row) wired EngineArgs -> CLI -> config, hash-covered (tests).
RowCacheWeightProvider: CachedWeightProvider surface plus NVFP4 global-scale slot buffers, owner-stream contract, forward-scoped map, LRU with in-forward victim protection, staged copies on a provider-owned copy stream (release event on the owner stream -> copy stream waits -> six-tensor copies -> ready event -> owner waits), invalidate ordered against in-flight copies. Expert maps are uploaded as one pinned non-blocking copy each; per-element device writes stalled the host until the owner stream drained (d26e289).
(b1) CachedWeightProvider: optional keyword-only w13_scale_2/w2_scale_2 with buf_*_scale_2 slot buffers; ExpertWeightResult and the path without them unchanged.
(c) RoutedExperts: provider selection by the knob; _maybe_init_expert_lru_cache(scale_suffix, *, scale_2_suffix) repoints per-expert global-scale parameters at slot buffers before the quant config is built; run_with_expert_cache typed against a Protocol.
Load path: with the cache enabled, ModelOptNvFp4FusedMoE.create_weights allocates the per-expert weights and block scales in CPU pinned memory; prepare_nvfp4_moe_layer_for_marlin converts host-resident sources expert_chunk (64) experts at a time on the accelerator into pinned host outputs (device sources: unchanged single pass); the providers alias the pinned results.
(b2) ModelOptNvFp4FusedMoE: supports_expert_lru_cache True only for the Marlin backend (expert-major final layout, globals read through the quant config); cache installed after the backend conversion, before quant config/kernel; apply through run_with_expert_cache; monolithic rejects a cache.
Loader interaction: device_loading_context restores CPU-resident parameters by name after processing; the cache marks its scale slot buffers device-resident (DEVICE_RESIDENT_ATTR) so they are not pulled back. Ordinary replacements keep the previous contract; the context no longer retains moved Parameter objects.
Breakable CUDA graphs: for models that are not torch-compiled, the MoE custom ops are eager break points for layers holding a provider (uncached MoE layers stay captured), and both guards accept VLLM_USE_BREAKABLE_CUDAGRAPH=1 as the alternative to compile splitting. The legacy layer-name placeholder is resolved once per op.
Byte accounting at cache setup and periodic row-provider stats.
Stream ownership in the row provider is tracked, not enforced: the release event is recorded on the stream of the previous accepted prepare() (where that forward read the slots) and the current stream becomes the owner after waiting on the ready event; resident-map writes are serialized with the slot copies (copy stream in prepare(), owner stream in invalidate()). Contract: sequential host calls, all consumer reads enqueued on the current stream before the next prepare().

Tests

CPU/mock: tests/config/test_moe_expert_cache_provider.py, tests/kernels/moe_provider/test_row_provider_cpu.py, test_row_provider_staging_mock.py (event call order).
CUDA: tests/kernels/moe_provider/test_row_provider_cuda.py (eviction reuse with a pre-captured graph reader kept unfinished while the next prepare is enqueued; previous reader output oracle-checked; six-tensor bytes on the reused slot; every route planned once; other-stream rejection; invalidate->prepare ordering), tests/kernels/moe/test_expert_lru_cache.py (scale_2 lifecycle/validation added), tests/kernels/moe/test_nvfp4_expert_cache_consumer.py (Marlin capability-gated; both providers x device/host sources; distinct per-expert globals; overflow every forward; per-pass g1/g2_alphas[slot] equal to the uncached layer's final globals of the mapped expert; live-buffer identity; output match; chunked host repack byte-equal to the device path for N=128 and N=96), tests/kernels/moe/test_nvfp4_marlin_repack_args.py (CPU: non-positive expert_chunk rejected before side effects).

Executed: Mac: ruff, format, mypy, pre-commit; CPU+mock provider tests (9 passed, run outside the repo conftest). GPU (astra, overlay on this base, head d26e289): tests/kernels/moe_provider + test_expert_lru_cache.py 51 passed. GPU on the exact a97 build (astra, head b2b5ec1): tests/kernels/moe_provider, test_expert_lru_cache.py, test_nvfp4_expert_cache_consumer.py, test_nvfp4_marlin_repack_args.py = 63 passed, 0 skipped (results/upstream-provider-exact-b2b5ec199799-20260909). Real checkpoint (astra, mid cross-checked): integration head lab/upstream-provider-ple-mmap (this PR merged onto lab/upstream-ple-mmap-base a7a3d7a = upstream/main a97dacb + proposed sync PLE vllm-project#54129; 4c79d36 for the row run, fcf878a for the cached run) on RTX PRO 6000 Blackwell Max-Q 96 GiB limited to 48 GiB by balloon; Marlin backend; --moe-expert-cache-size 208 --moe-expert-cache-split token; chunk 512; max_num_seqs 1; breakable CUDA graphs auto-on (compilation mode NONE), cudagraph_mode PIECEWISE, capture sizes [1, 2]; sync PLE mmap; no speculation. Both providers: load 35.54 GiB, 48 layers cached at 208/512, KV 9.86 GiB, captures at 1 and 2 tokens recorded twice each (profiling and actual capture; 98 graphs / 97 eager breaks per capture), two probe requests answered correctly (a 64-token explanation and a 5-token arithmetic answer 3288). Host cgroup peak reached the 100 GiB budget without OOM; capacity samples (25 at load, 4 during generation) recorded no error. Results: cached results/upstream-provider-checkpoint-fcf878ad5781-cached-20260909 (head 0b6fe94), row results/upstream-provider-checkpoint-4c79d366113b-row-20260909 (head b5ac54e), each with an independent review file. Exact suite at 4c79d36: 71 passed, 0 skipped. Limits: short smokes only; the servers were stopped externally after the requests and the logs keep the shutdown EngineDeadError/semaphore warnings (not a clean shutdown claim); no speed comparison between providers, no long-output quality, no per-step replay counters, no concurrent-consumer guarantee (sequential-consumer stream contract, see the class docstring).

Equal-capacity pair on the same head (astra, mid cross-checked; integration head e5c920b; same launch except --moe-expert-cache-provider; two fixed tokenized prompts A 1070 / B 753 tokens, max_tokens 2048, one request each, no repeats; results/upstream-provider-pair-e5c920b0fdda-20260909):

provider	A: output / TTFT / client decode	B: output / TTFT / client decode
cached	926 tok / 10.16 s / 18.24 tok/s	1616 tok / 6.36 s / 18.84 tok/s
row	878 tok / 9.39 s / 19.76 tok/s	1306 tok / 6.14 s / 19.66 tok/s

Single observations with differing outputs (greedy paths diverge between providers), so this is a check, not a speed claim: the row provider's staged copies show a few percent higher client decode rate at N=208. Both are far below the lab fork's post-fix figure (66 tok/s at a 26 GiB tier with the in-graph planner); that is a different configuration and not a single-variable comparison. Capacity samples recorded no guard error (cached 26 startup + 75 generation, row 27 + 62); host cgroup peak reached the 100 GiB budget in both runs (memory reclaim events 4012 / 4138) without OOM; both containers exited 0.

Port source for the internals: lab/flashnext-cpu-base 5bf2151 (global_pool, device_lru, staging, promote, device_record).

🤖 Generated with Claude Code

https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT

…ched) and RowCacheWeightProvider skeleton with the CachedWeightProvider surface - OffloadConfig.moe_expert_cache_provider wired through EngineArgs and the CLI; compute_hash covers it (test). - RowCacheWeightProvider: same surface as CachedWeightProvider plus per-expert global-scale buffers, stats counters, owner-stream contract, forward-scoped map. Internals in this head are pinned host source, device slots, LRU with protection of experts needed in the forward, synchronous copies; staged copies on a provider stream and boundary publication follow. Not wired into RoutedExperts yet. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…an_chunks, validate explicit ids before mutation, explicit device kwarg (review) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…ver alias), owner-stream check on invalidate, accurate copy wording (review) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…ate change (owner record and counters after validation) (review) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…ned copy stream with release/ready events, forward-generation map; CPU event-order mock and CUDA reuse/race tests Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…alidate ordering comment matches the contract, CUDA race test proves reader overlap over all six slot tensors, invalidate->prepare reuse test (review) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

… invalidate; CUDA tests state allocator behaviour accurately and prove reader overlap for invalidate->prepare too (review) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…race tests The previous reader (eager loops on the owner stream) could finish before prepare()/invalidate()+prepare() were enqueued, making the overlap assert inconclusive. The reader is now a pre-captured CUDA graph (finite GPU delay via preallocated matmuls, then a read of the real slot tensors) replayed on the owner stream; all buffers, events and the graph are prepared before the measured region, and the post-prepare `reader_done.query()` assert is kept. The graph is a fixture only, not evidence of graph support in the provider. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

Adds keyword-only `w13_scale_2`/`w2_scale_2` (given together, one row per expert) with pinned host copies and slot buffers filled on every miss, exposed as `buf_w13_scale_2`/`buf_w2_scale_2`. NVFP4 layers hand the slot buffers to the kernel in place of their parameters, so ExpertWeightResult is unchanged and the path without these tensors is untouched. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…ure is inconclusive Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

Per-element writes to the device map (`map[e] = slot`) stage a scalar through pageable host memory and block the host until the owner stream drains, i.e. until the previous reader finishes. That made the release-event ordering unobservable (found by a stage-by-stage GPU trace: query flipped at the first `self._map[victim] = -1`). The resident and forward maps are now built as host lists and uploaded with one pinned, non-blocking copy each; the caching host allocator keeps the staging tensor alive until the copy has consumed it. No behavior change for the CPU device. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…e_2 through slots Wires `OffloadConfig.moe_expert_cache_provider` ("cached" default, "row") into `_maybe_init_expert_lru_cache`, which now also accepts a `scale_2_suffix` for per-expert global scales: when present and expert-indexed they go to the provider and the layer parameters are repointed at the slot buffers before any quant config is built. `run_with_expert_cache` takes a Protocol so both providers type-check. A layer built without a model_config (unit tests) is treated as eager, matching the guard VllmConfig applies. The row provider's map upload docstring is narrowed to what it removes (scalar staging), per review. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…lin backend `supports_expert_lru_cache` is True only for MARLIN, whose final representation keeps all six expert tensors expert-major and reads the global scales through the quant config. After the backend conversion the layer installs the cache (source = final representation, parameters repointed at slot buffers, including `*_weight_scale_2`) and only then builds the quant config and kernel, so g1/g2_alphas are the live slot buffers. `apply` runs through `run_with_expert_cache` like FP8; the monolithic path rejects a cache. Test (CUDA, Marlin): both providers, experts with distinct global scales, three forwards that each overflow the cache; after every prepare() the consumer's own g1/g2_alphas per slot equal the uncached layer's final-representation globals of the expert mapped there, the arguments are the live buffers (data_ptr), and outputs match the uncached layer. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…kipping on any ValueError Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…mory, repack Marlin in chunks With the expert cache enabled, ModelOptNvFp4FusedMoE allocates w13/w2 weights and block scales in CPU pinned memory (as the unquantized method does), so loading needs no GPU capacity for them. prepare_nvfp4_moe_layer_for_marlin converts host-resident sources `expert_chunk` experts at a time on the accelerator and writes the results into pinned host tensors; the shared per-layer scale factor is still taken over all experts first. Device-resident sources take the previous single pass unchanged. The providers alias the pinned results (no second host copy). Tests: the chunked host path reproduces the device path byte-for-byte (3+3+2 experts); the Marlin consumer test also runs with host-resident sources on both providers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…chunk, alias contract wording, host/pinned checks in tests Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…d before side effects Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…P4 Marlin path's own predicate) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…s (checkpoint layout) ops.scaled_fp4_quant returns the swizzled 128x4 scale layout padded to 128 rows, so N=96 produced [E, 256, 16] scales and the Marlin pad view failed; the checkpoint contract is row-major [E, 2N, K/16]. Use the reference quantizer per expert and assert the shapes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…rovider stats The cache-enabled log line now reports slot-buffer bytes and the host source bytes; RowCacheWeightProvider.stats() gains slot_bytes/host_bytes and logs hits/misses/evictions/resident slots every 1000 prepares (INFO), matching the cached provider's interval. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

The context recorded CPU-resident parameters by name and, on exit, copied every parameter with those names back to pinned CPU memory. A quant method that replaces a parameter under the same name during processing (the expert cache repoints w13/w2 scale parameters at its device slot buffers) had the replacement's data moved to the CPU, and the kernel built from it then failed ("b_scales is not on GPU" from Marlin at the first forward). Track the moved Parameter objects and restore only those; replaced parameters keep their device data. Unreplaced CPU-resident parameters behave as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…only cache-marked slot buffers Review follow-up: restoring by Parameter identity changed the loader contract for every same-name replacement and kept the moved (already released) weights alive until the context exited. Names are recorded again, and the restore skips only parameters carrying DEVICE_RESIDENT_ATTR, which RoutedExperts sets on the scale/scale_2 slot buffers it installs. Ordinary repack replacements are moved back to the CPU as before. Tests cover both contracts and that a replaced weight is collectable inside the context. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…enerator frame holds no Parameter across the yield Review follow-up: the loop variable kept the last moved Parameter alive while the context was open, so a replaced weight could not be freed and the weakref regression would fail. The test docstring now states the narrowed contract and that the UVA branch is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

Models that default to breakable CUDA graphs are not torch-compiled, so the cache's "MoE op as a splitting op" requirement could not be met: the config guard rejected compilation mode NONE, and with VLLM_USE_BREAKABLE_CUDAGRAPH=0 capture failed (no piecewise path). The MoE custom ops are now eager break points under breakable graphs, only for layers that hold a provider, so prepare() runs eagerly between segments while uncached MoE layers stay captured; the output goes through the runner's existing capture-stable buffer. Both guards (VllmConfig, RoutedExperts) accept breakable mode as an alternative to VLLM_COMPILE splitting. Tests cover the dispatch. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

… in the break-point wrapper Review follow-up: get_layer_from_name is stateful for the legacy "from_forward_context" name (one forward-context entry per MoE op), and the wrapper plus the op each called it, consuming two layers per op. The stateful resolution is split out (resolve_concrete_layer_name), done once in the wrapper, and the concrete name is passed to the op. Regression: two legacy layers, cached and uncached, one entry consumed per call. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…nd checks thunk replay keeps the fixed name Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

… MoERunnerInterface Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…atch tests Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…jecting other streams Under breakable CUDA graphs prepare() runs on the profile stream, then the capture stream (eager break), then the replay stream, so the first-stream ownership rule rejected the capture-time call. The provider now records the release event on the stream of the previous accepted prepare() (where that forward's kernels read the slots), has the copy stream wait on it, and makes the current stream the new owner after it waits on the ready event; the ordering contract is unchanged, only the stream identity moves. invalidate() no longer touches ownership, so a later reuse is still ordered behind the last forward's reader. Stats expose owner_changes. Tests: CUDA reader on the previous owner stream kept unfinished while prepare() is issued from another stream (overlap asserted, old reader value and new slot bytes checked); other-stream prepare takes ownership; the CPU mock checks the release/ready targets across two owner changes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…he slot copies across owner changes Review follow-up: the shared resident map was uploaded on the current stream before it waited on the ready event, so after an owner change an upload queued on the previous stream (behind its reader, or from invalidate()) could land after the new stream's upload and revert the map to an older generation. prepare() now writes the resident map on the copy stream after the release wait (ordered behind the previous owner's reader and uploads, published by the ready event with the copies); invalidate() writes it on the owner stream, in program order behind the last reader, so the next prepare()'s release event orders it too. The forward map is unchanged (fresh tensor after the ready wait). Tests: CUDA regressions assert the resident map ends in the final generation after a cross-stream prepare and after invalidate() from another stream while the previous reader is still running; the CPU mock checks the invalidate write targets the owner stream. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

…the call's counts; host_bytes scope Review follow-ups (no behavior change beyond the log placement): the class docstring states the sequential-consumer stream contract; the periodic INFO line is emitted after the call's bookkeeping so its counts include the call and it says the host figure spans all six source tensors; host_bytes' docstring is conditional on the device. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

01554 added 30 commits September 9, 2026 11:08

tests: feedback-free GPU delay and distinct-sum precheck in race fixture

d54f3c8

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QWXP5rMj1rGh9xasXNLyT Signed-off-by: 01554 <24953377+01554@users.noreply.github.com>

01554 added 2 commits September 9, 2026 14:16

Navigation Menu

[Draft][MoE][Offload] Row-level expert weight cache provider (moe_expert_cache_provider=row) - #47

[Draft][MoE][Offload] Row-level expert weight cache provider (moe_expert_cache_provider=row)#47
01554 wants to merge 32 commits into
lab/upstream-37190-basefrom
lab/upstream-expert-row-provider

01554 commented Sep 9, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

01554 commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

01554 commented Sep 9, 2026 •

edited

Loading